Differential Temporal Difference Learning

Value functions derived from Markov decision processes arise as a central component of algorithms well performance metrics in many statistics and engineering applications machine learning. Computation the solution to associated Bellman equations is challenging most practical cases interest. A popular class approximation techniques, known temporal difference (TD) learning algorithms, are an important subclass general reinforcement methods. The introduced this article intended resolve two well-known issues with TD-learning algorithms. Their slow convergence due very high limit theorem variance, fact that, for problem computing relative value function, consistent exist only special cases. First we show that gradients these admit representation lends itself algorithm design. Based on result, new differential TD-learning introduced. For Markovian models Euclidean space smooth dynamics, shown be under conditions. Numerical results dramatic variance reduction comparison standard